Shared memory architecture

ABSTRACT

Disclosed herein is an apparatus which may comprise a plurality of nodes. In one example embodiment, each of the plurality of nodes may include one or more central processing units (CPUs), a random access memory device, and a parallel link input/output port. The random access memory device may include a local memory address space and a global memory address space. The local memory address space may be accessible to the one or more CPUs of the node that comprises the random access memory device. The global memory address space may be accessible to CPUs of all the nodes. The parallel link input/output port may be configured to send data frames to, and receive data frames from, the global memory address space comprised by the random access memory device(s) of the other nodes.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 11/807,986, filed on May 31, 2007, entitled “SHARED MEMORYARCHITECTURE,” which is hereby incorporated by reference.

TECHNICAL FIELD

This description relates to computing systems.

BACKGROUND

Computing systems may be built on a cluster of autonomous servers, andmay be connected to each other by switches, such as Ethernet switches.The servers may communicate with each other by sending and receivingmessages. It is desirable to reduce the latency and increase thebandwidth of computing systems.

SUMMARY

According to one general aspect, an apparatus includes a plurality ofnodes. Each of the plurality of nodes includes one or more centralprocessing units (CPUs), a random access memory device, and a parallellink input/output port. The random access memory device includes a localmemory address space and a global memory address space. The local memoryaddress space is accessible to the one or more CPUs of the node thatcomprises the random access memory device. The global memory addressspace is accessible to CPUs of all the nodes. The parallel linkinput/output port is configured to send data frames to, and receive dataframes from, the global memory address space comprised by the randomaccess memory device(s) of the other nodes.

According to another general aspect, an apparatus includes a pluralityof nodes and a switch for routing a frame between the nodes. Each of theplurality of nodes includes a central processing unit (CPU) for writingthe frame to a local memory address space, a memory divided between thelocal memory address space and a global memory address space associatedwith a global address, a protocol engine, and a parallel linkinput/output port. The protocol engine is configured to instruct theparallel link input/output port to prepend a routing header onto theframe stored in the local memory address space and send the frame out ofthe node through the parallel link input/output port. The protocolengine is also configured to receive the frame through the parallel linkinput/output port and write the frame to the global memory address spacebased on the routing header including the global memory addressassociated with the global memory address space. The parallel linkinput/output port is configured to prepend the routing header onto theframe, strip the routing header from the frame, and route the framebetween the protocol engine and the switch. The switch is configured toroute the frame between the plurality of nodes.

According to another general aspect, a system includes a plurality ofnodes. Each operates under an independent operating system and includesa memory including a local memory accessible to the node and a globalmemory accessible to all the nodes in the system.

According to another general aspect a memory manager is configured toload onto a operating system of each of a plurality of nodes andpartition a memory of each of the nodes into a local memory accessibleto the node and a global memory accessible to all of the plurality ofnodes.

Implementations can include one or more of the following features. Forexample, each node may include a bus. In another example, the parallellink input/output port may include a HyperTransport port or PCI Expressport.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features will beapparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram of an apparatus including a plurality ofnodes according to an example embodiment.

FIG. 1B is a block diagram showing one of the plurality of nodesaccording to another example embodiment.

FIG. 2 is a block diagram showing logical memory space and physicalmemory space of the apparatus of FIG. 1A according to an exampleembodiment.

FIG. 3A is a block diagram of a frame according to an exampleembodiment.

FIG. 3B is a block diagram of the frame with a routing header prependedonto the frame according to an example embodiment.

FIGS. 3C-3F are block diagrams of the routing header according to fourexample embodiments.

FIG. 3G is a block diagram showing a notification message.

FIG. 4 is a block diagram showing protocol levels which the frame maypass through or bypass.

DETAILED DESCRIPTION

FIG. 1A is a block diagram of an apparatus 100 including a plurality ofnodes 102A, 102B, 102C, and 102D according to an example embodiment. Theapparatus 100 may, for example, include a computing system, such as ahigh-performance computing system, which may utilize a shared logicalmemory space that is created from physical memory resources of the nodes102A, 102B, 102C, and 102D. Addresses in the shared logical memory spacemay be, for example, mapped to a global memory address space.

In the example embodiment shown in FIG. 1A, the apparatus 100 includesfour nodes 102A, 102B, 102C, and 102D. However, the apparatus 100 mayinclude any number of nodes, such as two nodes, four nodes, sixteennodes, et cetera. The apparatus 100 may also include a plurality of pods(not shown). Each pod may include a plurality of nodes 102. For example,the apparatus 100 shown in FIG. 1A may include a first pod that mayinclude a first node 102A of the first pod and a second node 102B of thefirst pod, and a second pod may include a first node and a second node.Other numbers of pods, as well as other numbers of nodes per pod,including pods each including a different number of nodes, may beincluded in other embodiments of the apparatus 100.

Each node 102A, 102B, 102C, and 102D may operate under an independentoperating system, such as Linux, according to an example embodiment.Each node 102A, 102B, 102C, and 102D may also be associated with an IPaddress and/or a medium access control (MAC) address, according to anexample embodiment. The IP address and/or the MAC address associatedwith each node 102A, 102B, 102C, and 102D may include a node ID and/or apod ID of the node. For example, a fifth node in a fourth pod may beassociated with an IP address such as ‘192.128.4.5,’ where ‘4’ is thepod ID associated with the pod which includes the node, and ‘5’ is thenode ID associated with the node. Similarly, the same node may beassociated with an Ethernet address such as ‘/OUIF45,’ where ‘4’ is thepod ID associated with the pod that includes the node, and ‘5’ is thenode ID associated with the node.

The apparatus 100 may be configured to route data, such as dataformatted into frames (discussed with reference to FIGS. 3A and 3B),between the nodes 102. The apparatus 100 may also be configured to sendand receive data from the apparatus 100 to devices outside the apparatus100, such as through a network (not shown). The apparatus 100 may, forexample, send and receive data through the Internet, using protocolssuch as TCP/IP, according to an example embodiment.

The apparatus 100 may, for example, include an Ethernet switch 104. TheEthernet switch 104 may be configured to route data between the nodes102. The Ethernet switch 104 may route data between the nodes 102according to any of various Ethernet protocols such as switchedEthernet, fast Ethernet, gigabit Ethernet, or 10 gigabit Ethernet. TheEthernet switch 104 may route data based on a MAC address or IP addresswhich includes the node ID, discussed above.

The apparatus 100 may include a parallel link switch 106. The parallellink switch 106 may utilize a parallel link technology utilizing aplurality of parallel data pins, such as HyperTransport or PCI Express,for example. The parallel link switch 106 may be configured to routedata between the nodes 102. The parallel link switch 106 may route thedata between the nodes according to a parallel link technology protocol,such as the HyperTransport protocol or PCI Express protocol, accordingto example embodiments. The parallel link switch 106 may route the data,which may be formatted into frames, between the nodes 102 based on arouting header prepended onto the frames. The routing header isdiscussed further with reference to FIGS. 3B-3F. According to an exampleembodiment, the parallel link switch 106 may replace the Ethernet switch104. In such an example, the apparatus 100 may include the parallel linkswitch 106 but not the Ethernet switch 104.

Each node 102 may include at least one central processing unit 108(CPU). Sixteen distinct CPUs are shown in FIG. 1A. CPUs 108A1, 108A2,108A3, and 108A4 are associated with node 102A. CPUs 108B1, 108B2,108B3, and 108B4 are associated with node 102B. CPUs 108C1, 108C2,108C3, and 108C4 are associated with node 102C. CPUs 108D1, 108D2,108D3, and 108D4 are associated with node 102D. For convenience, thesuffixes of the reference numerals that refer to the CPUs may, at times,be omitted.

As shown in FIG. 1A, each node 102A, 102B, 102C, and 102D can includefour CPUs 108. Each CPU 108 may be configured to perform operations ondata, such as writing frames to memory, for example. The CPUs 108 may beconfigured to write the frames to memory in an Ethernet format, such asby writing frames that include a header that includes a destination MACaddress and a source MAC address.

In an example embodiment, the CPUs 108, which may be considered to beexamples of a protocol engine, may retrieve data, such as data writtenin an Ethernet format by software (not shown), from local memory(discussed below), and may send the data to a port (discussed below)with instructions to prepend a routing header onto the frame (discussedwith reference to FIGS. 3B-3F). Or, the CPUs 108 may instruct anotherprotocol engine, such as a packet manager (discussed below), nodecontroller (discussed with reference to FIG. 1B), or a data mover (alsodiscussed with reference to FIG. 1B), to retrieve the data, which may bewritten in the Ethernet format, from local memory, and send the data tothe port with instructions to prepend the routing header onto the frame.Instructing another protocol engine to perform these tasks may reducethe work load on the CPUs 108.

Each node 102A, 102B, 102C, and 102D may include a random access memory(RAM) device 110A, 110B, 110C, and 110D, respectively, coupled to theCPUs 108 that are associated with the node. Each RAM 110A, 110B, 110C,and 110D may be partitioned into a global memory address space 112A,112B, 112C, and 112D and a local memory address space 114A, 114B, 114C,and 114D, respectively. The local memory address space 114A of the firstnode 102A may be accessible only to the CPUs 108A1-4 of only the firstnode 102A, whereas the global memory address space 112A of the firstnode 102A may be accessible to the all sixteen CPUs 108 of the nodes102A, 102B, 102C, and 102D included in the apparatus 100.

The global memory address space 112A of the first node 102A may be madeaccessible to all sixteen CPUs 108 by loading a distributed memorymanager into the operating system or kernel as a driver. Each globalmemory address space 112A, 112B, 112C, and 112D may be associated with aglobal memory address, according to an example embodiment. The globalmemory address space 112A, 112B, 112C, and 112D may be configured toreceive data routed to respective node 102A, 102B, 102C, and 102D whichincludes the global memory address. According to this embodiment,messages sent over a network may be routed to the global memory addressspace 112 based on the global memory address.

The physical memory address spaces of the RAM 110 of each node 102 maybe divided into the global memory address space 112 and the local memoryaddress space 114. The local memory address space 114 of a node may, forexample, include addresses from 0 to 0×00_FFFF FFFF for sixty-fourgigabytes of memory. The global memory address space 112 may includeaddresses prefixed with the node ID of the node 102, such as0×N0_(—)0000_(—)0000 for node N. The global or shared memory addressesfor all nodes 102 in the apparatus 100 may range from0×N0_(—)0000_(—)0000 to 0×M0_(—)0000_(—)0000, in the example where N, Mrange from zero to fifteen for the example of sixteen nodes 102. Node Ncould access that portion of the shared memory contributed by node M byissuing a request (or read command, discussed with reference to FIGS. 3Eand 3F) with the address in the range of 0×M0_(—)0000_(—)0000 to0×M0_FFFF_FFFF, according to an example embodiment. The global or sharedmemory address may identify a physical region or buffer within theglobal memory space 112. The pods, which each include a plurality ofnodes 102, may or may not share the global memory with each other.

Each node 102 may also include a packet manager 116 coupled to the CPUs108, to the RAM 110, and to a parallel link port 118 associated with thenode. The packet manager 116 may be an example of a protocol engine. Forexample, node 102A may include a packet manager 116A coupled to the CPUs108A1, 108A2, 108A3, and 108A4, to the RAM 110A, and to a parallel linkport 118A. The packet manager 116 for a node may include a direct memoryaccess (DMA) engine, according to an example embodiment. The packetmanager 116 of a node may, for example, retrieve, from the local memoryaddress space 114 of the node, data frames that were written to thelocal memory address space 114 by a CPU 108 of the node. The data mayhave been written to the local memory address space 114 in an Ethernetformat by software, according to an example embodiment.

According to an example embodiment, the packet manager 116 of one node102, such as the first node 102A, may be configured to instruct aparallel link input/output port to prepend a routing header onto a dataframe, and thereby write the frame through the parallel linkinput/output port 118A to the global memory address space (e.g., 112B,112C, or 112D) that is physically located in a random access memory(e.g., 110B, 110C, or 110D) of another node (e.g., 102B, 102C, or 102D).The packet manager 116A may, for example, write the frame to the globalmemory address space 112B, 112C, or 112D that is physically located inthe random access memory device 110B, 110C, or 110D of another node102B, 102C, or 102D based on the node ID associated with the other node102B, 102C, or 102D included in the routing header of the frame, orbased on the frame including a global memory address.

According to another example embodiment, the packet manager 116 mayretrieve data, such as data written by software in an Ethernet format,from the local memory 114, and send the data to a parallel link port(discussed below) with instructions to prepend a routing header(discussed with reference to FIGS. 3B-3F) onto the data. The packetmanager 116 may perform these tasks in response to an instruction fromthe CPU 108, according to an example embodiment.

Each node 102 may also include the parallel link input/output port 118coupled to the packet manager 116 and the parallel link switch 106,according to an example embodiment. For example, node 102A can include aparallel link input/output port 118A. The parallel link port 118 of anode may route frames between the packet manager 116 of the node and theparallel link switch 106. The parallel link port 118 may utilize aparallel link technology, such as multiple pins for multiple paralleldata channels, according to an example embodiment. The parallel linkport 118 may be configured to send and receive data, such as dataformatted into frames, from the global memory address space 112comprised by the RAM 110 of other nodes 102. The parallel link port 118may include a Hypertransport port or a PCI Express port, according toexample embodiments.

The parallel link port 118 may prepend a routing header onto dataaccording to instructions from the CPU 108, packet manager 116, datamover (discussed below), or node controller (discussed below). Theparallel link port 118 may also frame the data. The parallel link port118 may then send the frame to the parallel link switch 106, accordingto an example embodiment. The parallel link port 118 may also receiveframes from the parallel link switch 106, strip the routing header fromthe frame, and send the frame to the node controller, data mover, packetmanager 116, CPU 108, local memory 114, or global memory 112, accordingto example embodiments. After the frame has been written to the localmemory 114 or global memory 112 in Ethernet format after having therouting header stripped from the frame, the frame may appear to thesoftware to have been routed according to Ethernet protocols.

The parallel link port 118 of a node may facilitate the receipt of datafrom other nodes 102. For example, the parallel link port 118 of a nodemay receive a frame that is sent by another node 102, strip a prependedrouting header from the frame, and route the frame to the global memoryaddress space 112 of the RAM 110 comprised by the node 102 in which theparallel link port resides if the prepended routing header includes aglobal memory address associated with the global memory address space112 of the RAM 110 comprised by the node 102. If the prepended routingheader does not include an address associated with the global memoryaddress space 112, the packet manager 116 may discard the frame,according to an example embodiment.

If the frame is written to the local memory address space 114A of node102A by the CPU 108A in an Ethernet format, then the frame, after havingthe routing header stripped by the parallel link port 118B, 118C, or118D of another node 102B, 102C, or 102D and routed to the global memoryaddress space 112B, 112C, or 112D of the other node, may appear to theCPU 108B, 108C, or 108D or software of the other node to be an Ethernetframe. The packet managers 116A, 116B, 116C, or 116D, parallel linkports 118A, 118B, 118C, or 118D, and parallel link switch 106 maythereby be used to emulate Ethernet protocols by transmitting framesbetween the nodes which appear to the CPUs 108 to be Ethernet frames.However, embodiments described herein and their variants may provideshorter latency and higher bandwidth for sending frames between nodes102 than Ethernet devices.

Each node 102 may also include a buffer 120. For example, node 102A caninclude a buffer 120A. The buffer 120 may be coupled to the CPUs 108,the RAM 110, the packet manager 116, and an Ethernet port 122 of thenode. The buffer 120 may be configured to act as an intermediary betweenthe Ethernet port 122 and the CPU 108 and/or the RAM 110 of the node.

Each node 102 may also include the Ethernet port 122. For example, node102A can include an Ethernet port 122A. The Ethernet port 122 of a nodemay be coupled to the buffer 120 and to the Ethernet switch 104 of thenode. The Ethernet port 122 may be configured to route data, such asdata formatted into Ethernet frames, into and out of the node 102, suchas between the node 102 and the Ethernet switch 104. The Ethernet port122 may utilize any of various Ethernet protocols such as switchedEthernet, fast Ethernet, gigabit Ethernet, or 10 gigabit Ethernet,according to example embodiments.

According to another example embodiment, a node 102 may not include abuffer 120, and the Ethernet port 122 of the node may be coupled to theEthernet switch 104, the CPUs 108, the RAM 110, and the packet manager116 of the node.

A node 102 may also include a bus 124. For example, node 102A caninclude a bus 124A. The CPUs 108, the RAM 110, the packet manager 116,and the buffer 120 of the node may be coupled to each other via the bus124. The bus 124 may facilitate routing data between the CPUs 108, theRAM 110, the packet manager 116, and the buffer 120 of a node, accordingto an example embodiment.

In an example embodiment, the parallel link switch 106 may include aglobal port 126. The global port 126 may be configured to receive dataframes based on a global address included in the frames that correspondsto a global address of one of the global memory address spaces 112. Theglobal port 126 of the parallel link switch 106 may be connected to anetwork, according to an example embodiment. In this example, theparallel link switch 106 may be configured to route a received frame toone of the nodes 102A, 102B, 102C, and 102D based on a global memoryaddress included in the frame that corresponds to a global addressassociated with the node 102A, 102B, 102C, and 102D.

FIG. 1B is a block diagram showing details of a node 102, according toanother example embodiment. The node 102 may include a Broadcom (BCM)1480 chip, according to an example embodiment. In this example, theplurality of CPUs 108X1, 108X2, 108X3, and 108X4 and the packet manager116 may be coupled to the bus 124. The CPUs 108X1, 108X2, 108X3, and108X4 may include L1 caches and instruction caches. The bus 124 mayinclude a low-swing differential ZB bus, according to an exampleembodiment.

This example may also include a cache 128 coupled to the bus 124. Thecache 128 may include an L2 cache which may function as a front-endcache to the RAM 110 (not shown in FIG. 1B). The cache 128 may allocatespace for received data according to a controllable allocation policy.

The node 102 may also include a DRAM controller 130 coupled to the bus124. The DRAM controller 130 may control access to, and functionsperformed by, the RAM 110. The DRAM controller 130 may be coupled to theRAM 110, according to an example embodiment.

The node 102 may also include a data mover 132 coupled to the bus. Thedata mover 132 may be an example of a protocol engine, and may include aDMA engine, such as a versatile high-performance four-channel DMAengine. The data mover 132 may free the CPUs 108X1, 108X2, 108X3, and108X4 from moving data between source and destination locations in theglobal memory address space 112 and/or the local memory address space114. The data mover 132 may enable data transfer between input/outputdevices and between nodes 102. The data mover 132 may thereby enabledata to be copied to a remote device's memory by specifying the remoteuser. The data mover 132, which may include a TCP offload engine in anexample embodiment, may also support data-touching functions bycalculating a checksum and cyclic redundancy code digest for arbitraryprotocol data units. This may relieve the CPUs 108X1, 108X2, 108X3, and108X4 from processing data, according to an example embodiment.

For example, the data mover 132 may retrieve data from the global memoryaddress space 112 or local memory address space 114 and send the data tothe parallel link port 118 with instructions to prepend a routing headeronto the data. As discussed above, the data may have been written tomemory by software in an Ethernet format. The data mover 132 may performthese tasks in response to instructions from the CPU 108, according toan example embodiment. The data mover 132 may also receive a frame fromthe parallel link port 118 and write the frame to the memory, accordingto an example embodiment. According to another example embodiment, theparallel link port 118 may write the frame directly to memory inEthernet format; in either example, the software may see only theEthernet frame, and the software may view the data as being sentaccording to Ethernet protocols.

In the example shown in FIG. 1B, the node 102 may include a pluralityof, such as three, parallel link ports 118. The parallel link ports 118each may be coupled to a switch 134, which in turn may be coupled to thepacket manager 116 and to a node controller 136. The parallel link ports118 may also be coupled to the parallel link switch 106 (shown in FIG.1A).

The node controller 136 may be coupled to the switch 134 and to the bus124; in an example embodiment, the node controller 136 may be combinedwith the packet manager 116, and the combination may be referred to asthe packet manager 116. The node controller 136 may handle input-outputaccess through the parallel link ports 118, according to an exampleembodiment. The node controller 136 may implement a write-invalidatedirectory-based protocol to support cache coherence among multiple nodes102. The node controller 136 may instruct the node 102 to monitor onlythose local memory lines that are cached remotely. The node controller136 may achieve this limited monitoring by using a directory such as aremote line directory (RLD). The RLD entry may be allocated for a memoryline when a remote node 102 requests the memory line, and may bede-allocated when all remotely cached copies are invalidated, accordingto an example embodiment. Each RLD entry may contain a vector ofpresence bits corresponding to each node 102 in the apparatus 100. Eachpresence bit may indicate whether the corresponding node 102 has a copyof a memory block (each copy may be identical to the copy in theoriginating node 102). The RLD entry may also include dirty bitsindicating whether a dirty cached copy exists. If a dirty cached copyexists, the protocol may require that only one node hold a cached copy,and may achieve this by sending invalidations to nodes 102 whichcorrespond to bits indicating the presence of dirty cached copies.

The coherence protocol may include a four-leg protocol, according to anexample embodiment. In response to a request for a copy along a memoryline, the node 102 with the copy may write a dirty line back to a homenode 102, and the home node 102 may forward the data to the requestingnode 102. The home node 102 may collect acknowledgments to invalidationrequests and provide data to the requesting node 102 upon completion.

The node 102 may also include an input/output bridge 138 coupled to thebus 124 and to DMA channels 140. The DMA channels 140 may be coupled tothe Ethernet port 122. The DMA channels 140 may include a plurality oftransmit and receive DMA channels, according to an example embodiment.The DMA channels 140 may transfer packets to memory, according to anexample embodiment. The Ethernet port 122 may be coupled to the Ethernetswitch 104 (shown in FIG. 1A).

FIG. 2 is a block diagram showing relationships between logical memoryspaces 202 and physical memory spaces 204 of the apparatus 100 of FIG.1A according to an example embodiment. As shown in the exampleembodiment of FIG. 2, the global memory may be mapped into a number oflogical levels equal to the number of nodes 102. For example, the firstnode 102A may contribute its global memory address space 112A to a firstlevel of the logical memory space 202, the second node 102B maycontribute its global memory address space 112B to a second level of thelogical memory space 202, the third node 102C may contribute its globalmemory address space 112C to a third level of the logical memory space202, and the fourth node 102D may contribute its global memory addressspace 112D to a fourth level of the logical memory space 202. The localmemory address spaces 114A, 114B, 114C, and 114D may be reserved by therespective nodes 102A, 102B, 102C, and 102D, and not contributed to theglobal memory.

The global memory address space 112A, 112B, 112C, and 112D mapped in thelogical memory space 202 may correspond to the global memory addressspace 112A, 112B, 112C, and 112D included in the physical memory space204. The physical memory space 204, which may include both the globalmemory address space 112A, 112B, 112C, and 112D and the local memoryaddress spaces 114A, 114B, 114C, and 114D, may be included in each ofthe nodes 102A, 102B, 102C, and 102D (shown in FIG. 1A), according to anexample embodiment.

The global memory address space 112 may be managed by a distributedmemory manager, according to an example embodiment. The distributedmemory manager may be loaded onto the operating system as a driver anddesignated with a unique node ID. The global memory may thereby beaccessed by the global memory address, such as the global memory addressdiscussed in paragraph [0029].

According to an example embodiment, the local memory address spaces114A, 114B, 114C, and 114D may be known to and available to theoperating system, such as the Linux kernel, of the node 102A, 102B,102C, and 102D that includes the particular local memory address space.The global memory address space 112, however, may be hidden to theoperating system, such as the Linux kernel, of the node 102 whichincludes the global memory address space 114; the physical associationbetween the global memory address space 112 and the node 102 withinwhich the global memory address space is included may thereby be unknownto the operating system of the node 102, making the global address space112 within the node 102 indistinguishable (other than by the globaladdress) to the operating system of the node 102 from the global addressspace 112 of other nodes 102 or global address spaces outside theapparatus 100.

While four levels of logical memory space 202 are shown in FIG. 2, thelogical memory space 202 may include other numbers of levels in otherembodiments. The number of levels of logical memory space 202 maycorrespond to the number of nodes 102 in the apparatus 100.

FIG. 3A is a block diagram of a frame 302 according to an exampleembodiment. The example frame 302 shown in FIG. 3A may have a format ofan Ethernet frame. The frame 302 may have been written to the localmemory address space 114 by a CPU 108 using software, and may bereceived by the software.

The example frame 302 shown in FIG. 3A includes a header 304, a framebody 306, and a frame check sequence (FCS) 308. The header 304 mayinclude routing information, such as a source MAC address 310 and adestination MAC address 312, according to an example embodiment. Theframe body 306 may include data to be transmitted from one device, suchas a node 102, to another device. The FCS 308 may include errordetection data, such as a checksum of the previous bits in the frame302.

FIG. 3B is a block diagram of the frame 302 with a routing header 314prepended onto the frame 302 according to an example embodiment. Therouting header 314 may have been prepended onto the frame 302 by thepacket manager 116, according to an example embodiment; the packetmanager 116 may also strip the routing header 314 off of a receivedframe 302. According to another example embodiment, the routing header314 may have been prepended onto the frame 302 by the parallel link port118; the parallel link port may have prepended the routing header 314onto the frame 302 according to instructions from a protocol engine suchas the CPU 108, packet manager 116, or data mover 132. The parallel linkport 118 may also strip the routing header 314 off of the frame 302. Therouting header 314 may include routing data which may be used by theparallel link switch 106 to route the frame 302 according to parallellink technology protocols, such as the HyperTransport protocol or thePCI Express protocol, according to example embodiments.

FIGS. 3C-3F are block diagrams of the routing header 314 according tofour example embodiments. The routing header 314 may include a pluralityof fields, such as a destination node ID 316, a command type 318, and acommand-sensitive field(s) 320, according to an example embodiment.According to an example embodiment, a selected one of a plurality, suchas four, types of routing headers 314A, 314B, 314C, 314D may beprepended onto the frame 302. The type of routing header 314 may bedetermined by the command type, according to an example embodiment. Thecommand types may include packet, write, read, and read response,according to an example embodiment.

The destination node ID field 316 may include the node ID of the node102 to which the frame 302 is to be sent. The parallel link switch 106may route the frame 302 to the destination node 102 based on dataincluded in the destination node ID field 316, according to an exampleembodiment.

The command type field 318 may include a command type, such as packet,write, read, or read response, according to an example embodiment. Thecommand type may be determined by the protocol engine, and the commandtype field 318 may be filled by the parallel link port 118 according toinstructions from the protocol engine, according to an exampleembodiment. The command sensitive field 320 may include fields which aredependent upon the command type.

FIG. 3C shows a packet routing header 314A, according to an exampleembodiment. The packet routing header 314A may be prepended onto a frame302 which is to be sent to a node 102 without instructions to write theframe 302 to a particular region of the global memory address space 112.The command type field 318A may indicate a packet. The command-sensitivefield 320A may indicate a payload length. The packet routing header 314Amay be prepended onto the frame 302 by the parallel link port 118according to instructions from the packet manager 116, according to anexample embodiment.

In this example, the frame 302 with a packet routing header 314A may bereceived by the receiving node 102, and the receiving node 102 maydecide where to store the frame 302. The parallel link port 118 of thereceiving node 102 may, for example, strip the packet routing header314A from the frame 302, and send the frame 302 to the packet manager116 based on the command type field 318A indicating that the header 302is a packet routing header 314A, according to an example embodiment. Thepacket manager 116 may have maintained a list of addresses of buffers orregions of memory in either the global memory address space 112 or localmemory address space 114, according to an example embodiment. The packetmanager 116 may write the received frame 302 to a buffer or region ofmemory in the list. The packet manager 116 may invoke the softwaredriver, which may pass the frame 302 up from the buffer or region ofmemory up through a TCP/IP protocol stack. The software may only seeEthernet frames, causing the software to view the data as being routedaccording to Ethernet protocols.

In another example embodiments, the tasks described above as performedby the packet manager 116 may be performed by the CPU 108 or the datamover 132.

FIG. 3D shows a write routing header 314B, according to an exampleembodiment. The write routing header 314B may be prepended onto a frame302 which is to be sent to a node 102 with instructions to write theframe 302 to a particular region of the global memory address space 112.The command type field 318B may indicate a write command. Thecommand-sensitive field 320B may include an address field 320B1 and adata field 320B2. The address field 320B1 may include a global memoryaddress of the receiving node 102. The data field 320B2 may include datato be written to the global memory address space 112 of the receivingnode 102.

The write routing header 314B may be prepended onto the frame 302 by theparallel link port 118 according to instructions from the data mover 132or the CPU. The parallel link switch 106 may route the frame 302 toanother node 102 based on a node ID included in the destination node IDfield 316. The parallel link port 118 of the receiving node 102 maystrip the write routing header 314B from the frame 302, and may pass thewrite command and the data included in the data field 320B2, or theframe 302 without the write routing header, to the node controller 136or packet manager 116 of the receiving node 102. The node controller 136may write the data to the portion of the global memory address space 112designated by the global memory address included in the address field320B1. The software driver may read the data as an Ethernet frame,causing the software to view the data as being routed according toEthernet protocols. The write routing header 314B may allow data to bewritten to a node 102 according to a “push” model, according to anexample embodiment.

FIG. 3E shows a read routing header 314C, according to an exampleembodiment. The read routing header 314C may be prepended onto a frame302 which is to be sent to a node 102 with instructions to write a frame302 to a particular region of the global memory address space 112 of thenode 102 which sent the read routing header 314C. The command type field318C may indicate a read command. The command-sensitive field 320C mayinclude an address field 320C1 and a transaction ID field 320C2. Theaddress field 320C1 may include a global memory address from which datashould be fetched by the node 102 which receives the read routing header314. The transaction ID field 320C2 may include a transaction ID whichis used to match data sent with a read response routing header 314D tothe read routing header 314C to which the data were sent in response.

FIG. 3F shows the read response routing header 314D, according to anexample embodiment. The read response routing header 314D may beprepended onto a frame which is sent to a node 102 in response to aframe with a read routing header 314C. The command type field 318D mayindicate a read response command. The command-sensitive field 320D mayinclude a transaction ID field 320D1 and a data field 320D2. Thetransaction ID field 320D1 may include a transaction ID which is used tomatch the read response routing header 314D and associated frame 302 tothe read routing header 314C and associated frame 302 which the readresponse routing header 314D and associated frame 302 are sent inresponse to. The data field 320D2 may include data read from the globalmemory address space 112 by the node 102 which received the read routingheader 314C based on the address 302C1.

FIG. 3G is a block diagram showing a notification message 322. Thenotification message 322 may include a destination node ID field 324which may include a destination node ID used by the parallel link switch106 to route the notification message 322 to a destination node 102B,and an address field 326 which may include an address of data in theglobal memory address space 112A of the sending node 102A which thesending node 102A is instructing the receiving node 102B to pull fromthe sending node 102A. The notification message 322 may also include alength field 328. The length field 328 may indicate how much data shouldbe read, such as a specified number of bytes.

The notification message 322, read routing header 314C, and readresponse routing header 314D may be used in a “pull” model in which dataare read from the global memory address space 112A of the node 102Awhich initiated the “pull”. The node 102A may initiate the pull bysending a notification message 322 to a second node 102B. According toan example embodiment, the first node 102A may send the notificationmessage 322 to the second node 102B. The notification message 322 mayinclude a node ID of a second node 102B in the destination node ID field324. The notification message 322 may also include, in the address field326, a global memory address of data which the first node 102A isinstructing the second node 102B to read. The notification message 322may also include, in the length field 328, the number of bytes (or othermeasure of data) which should be pulled. The value in the length field328 may be added to the global memory address in the address field 326to determine an ending address of data which should be pulled; in anexample embodiment, the pull operation may pull all of the data withglobal memory addresses between the global memory address included inthe address field 326 and this address plus the value in the lengthfield 328. All of these data may be pulled in one operation, or they maybe pulled in multiple, smaller operations.

According to an example embodiment, the second node 102B may respond tothe notification message 322 by sending a read frame to the first node102A. For example, a protocol engine, such as CPU 108B or data mover132B of the second node 102B may retrieve a frame 302 from memory, andsend the frame 302 to the parallel link port 118B with instructions toprepend a read routing header 314C onto the frame 302. The parallel linkport 118B may prepend the read routing header 314C onto the frame 302;the destination node ID field 316 may include a node ID of the firstnode 102A, and the address field 320C1 may include a global memoryaddress of the global memory address space 112A of the first node 102Abased on instructions from the address included in the address field 326of the notification message 322 sent by the first node 102A. Theparallel link port 118B may send the prepended frame 302 to the parallellink switch 106, and the parallel link switch 106 may send the prependedframe 302 to the first node 102A based on the node ID included in thenode destination node ID field 316 of the read routing header 316C.

The first node 102A may receive the prepended frame 302, and inresponse, may send a frame 302 with a read response routing header 314Dprepended onto the frame 302. A protocol engine, such as the CPU 108A ordata mover 132A, of the first node 102A, may retrieve data from memory,such as a frame 302 in an Ethernet format. The protocol engine may sendthe data or frame to the parallel link port 118A along with instructionsto prepend a read response routing header 314D onto the frame 302. Theparallel link port 118A may prepend the read response routing header314D onto the frame 302, and send the prepended frame 302 to theparallel link switch 106. The transaction ID field 320D1 may include thesame transaction ID as was included in the transaction ID field 320C1 ofthe read routing header 314C, based on the instructions received fromthe protocol engine, and the data field 320D2 may include data retrievedfrom the global memory address space 112A of the first node 102A basedon the address included in the address field 326 and the length includedin the length field 328 of the notification message 322. In the case ofmultiple pulls, the data included in a read response routing header 314Dmay be identified based on the transaction ID, which may be an identicaltransaction ID in both the transaction ID field 320C2 of the readrouting header 314C and the transaction ID field 320D1 of the readresponse routing header 314D; in an example embodiment, in the case ofmultiple read commands and read response commands, the data may be readfrom the appropriate portion of the global memory address space 112Abased on matching the transaction IDs included in the read routingheaders 314C and the read response routing headers 314D.

The parallel link switch 106 may send the prepended frame 302 to thesecond node 102B based on the node ID included in the destination nodeID field 316 included in the read response routing header 314D. Theparallel link port 118B of the second node 102B may strip the readresponse routing header 314D from the frame 302 and, based on the readresponse indicated by the command type field 314D, send the frame 302 tothe node controller 136B or packet manager 116B of the second node 102B.The node controller 136B or packet manager 116B may write the dataincluded in the data field 320D2 to a portion of either the globalmemory address space 112B or the local memory address space 114B. Thesecond node 102B may associate the data included in the data field 320D2with the prepended frame 302 that included the read routing header 314Cbased on the transaction ID included in the transaction ID field 320D1matching the transaction ID included in the transaction ID field 320C2of the read routing header 320C.

FIG. 4 is a block diagram showing protocol levels which the frame 302may pass through or bypass. The protocol levels may be divided into auser space 402 and a kernel space 404, according to an exampleembodiment.

The user space 402 may include a user processes layer 408. A messagepassing interface (MPI) library 410 may be included in both the userspace 402 and the kernel space 404. The kernel space 404 may alsoinclude a socket layer 412, a TCP/IP layer 414, and a packet manager(PM) driver layer 416.

A frame 302 sent from, for example, the first node 102A to the secondnode 102B may, for example, originate in the user processes layer 408A.The frame 302 may proceed from the user processes layer 408A through theMPI library 410A to the socket layer 412A. By use of the packet manager416A and the parallel link port 118A, the frame 302 may bypass theTCP/IP layer 414A, and proceed through the PM driver 416A to theparallel link port 406, according to an example embodiment. The frame302 may travel from the parallel link port 406 through the PM driver416B of the second node 102B, and bypass the TCP/IP layer 414B to thesocket layer 412B. The frame 302 may travel from the socket layer 412Bthrough the MPI library 410B to the user processes layer 408B, accordingto an example embodiment.

According to an example embodiment, a memory manager, such as adistributed memory manager, may be loaded onto each of a plurality ofkernels or operating systems of nodes 102 as a driver. The memorymanager may partition the memory 110 of each of the nodes 102 into alocal memory 114 accessible only to the node 102 of which the memory 110is a component, and a global memory 112 accessible to all the nodes 102.The memory manager may allow each node 102 to read data directly from,or write data directly to, the global memory 112 of the other nodes 102.

According to an example embodiment, the memory manager may allow eachnode 102 to send a packet, such as a frame 302 with a packet routingheader 314A, to the other nodes 102. In this example, the nodes 102 may,when receiving the packet, determine a portion of their memory in whichto store the packet. This determination of where to store the packet maybe made independently of information included in a header of the packet.

Also according an example embodiment, the memory manager may allow thenodes 102 to participate in a push process. The push process may includea first node 102A sending a write frame, such as a frame 302 with awrite routing header 314B, to a second node 102B. The write frame mayinclude a routing header 314B which includes an address and data. Thesecond node 102B may receive the write frame from the first node 102Aand store the data in a portion of the second node's 102B global memory112 based on the address included in the routing header 314B.

According to an example embodiment, the memory manager may allow thenodes to participate in a pull process. The pull process may include afirst node 102A sending any number of notification messages 322 to asecond node 102B. Each notification message 322 may include an addressin an address field 326. The second node 102B may receive thenotification message(s) 322. In response to receiving the notificationmessage(s) 322, the second node 102B may send a corresponding number ofread frame(s), such as frame(s) 302 with read routing header(s) 314C, tothe first node 102A. A header 314C of the read frame may include anaddress, such as in an address field 320C1, and may also include acorresponding transaction ID, such as in a transaction ID field 320C2.

The first node 102A may receive the read frame(s). In response toreceiving the read frame(s), the first node 102A may send acorresponding number of read response frame(s), such as frame(s) 302including a read response routing header 314D, to the second node 102B.The read response frame may include data stored in a portion of thefirst node's global memory based on the address included in the readframe. In the case of a plurality of read response frames, the readresponse frames may each include the transaction ID in a transaction IDfield 320D2, and the data included in the read response frame may bestored in a portion of the first nodes 102A global memory based on theaddress corresponding to the transaction ID. The second node 102B mayreceive the read response frame from the first node 102A and store thedata in the memory 110B of the second node.

Implementations of the various techniques described herein may beimplemented in digital electronic circuitry, or in computer hardware,firmware, software, or in combinations of them. Implementations mayimplemented as a computer program product, i.e., a computer programtangibly embodied in an information carrier, e.g., in a machine-readablestorage device or in a propagated signal, for execution by, or tocontrol the operation of, data processing apparatus, e.g., aprogrammable processor, a computer, or multiple computers. A computerprogram, such as the computer program(s) for use with the methods andapparatuses described above, can be written in any form of programminglanguage, including compiled or interpreted languages, and can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program can be deployed to be executed on onecomputer or on multiple computers at one site or distributed acrossmultiple sites and interconnected by a communication network.

Method steps may be performed by one or more programmable processorsexecuting a computer program to perform functions by operating on inputdata and generating output. Method steps also may be performed by, andan apparatus may be implemented as, special purpose logic circuitry,e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. Elements of a computer may include atleast one processor for executing instructions and one or more memorydevices for storing instructions and data. Generally, a computer alsomay include, or be operatively coupled to receive data from or transferdata to, or both, one or more mass storage devices for storing data,e.g., magnetic, magneto-optical disks, or optical disks. Informationcarriers suitable for embodying computer program instructions and datainclude all forms of non-volatile memory, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory may be supplemented by, or incorporated in special purposelogic circuitry.

Implementations may be implemented in a computing system that includes aback-end component, e.g., as a data server, or that includes amiddleware component, e.g., an application server, or that includes afront-end component, e.g., a client computer having a graphical userinterface or a Web browser through which a user can interact with animplementation, or any combination of such back-end, middleware, orfront-end components.

While certain features of the described implementations have beenillustrated as described herein, many modifications, substitutions,changes and equivalents will now occur to those skilled in the art. Itis, therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the true spiritof the embodiments of the invention.

1. An apparatus comprising: a plurality of processing nodes, each nodeincluding: one or more central processing units (CPUs); a random accessmemory device including: a local memory address space, accessible to theone or more CPUs of only the node that includes the random accessmemory; and a global memory address space, accessible to CPUs of all thenodes; a parallel link input/output port configured to send data framesto, and receive data frames from, the global memory address spacecomprised by the random access memory device(s) of other nodes; and aprotocol engine coupled to the parallel link input/output port, whereinthe protocol engine is configured to instruct the parallel linkinput/output port to prepend a routing header onto a medium accesscontrol (MAC) frame, the routing header including a destination addressand a command type.
 2. The apparatus of claim 1, wherein the commandtype of the routing header includes one of packet, write, read, or readresponse.
 3. The apparatus of claim 1, wherein: the command type ispacket; the parallel link input/output port is configured to strip therouting header from the MAC frame based on the command type beingpacket; and the node is configured to write the MAC frame to its randomaccess memory based on the command type being packet.
 4. The apparatusof claim 1, wherein the command type is write; wherein the routingheader further includes an address field designating a portion of theglobal memory address space, and a data field including data; andwherein the node is configured to write the data included in the routingheader to the portion of its global memory address space designated bythe address field of the header based on the command type being write.5. The apparatus of claim 1, wherein the routing header includes thedestination address and an address field including a global memoryaddress; wherein the command type is read; and wherein the node isconfigured to fetch data from the global memory address included in theaddress field based on the command type being read.
 6. The apparatus ofclaim 1, wherein the routing header includes the destination address,the command type includes a read response type; wherein the routingheader includes data read from the global memory address space, wherethe data was sent by a node that prepended the read response commandtype to a frame that included the data; and wherein the data included inthe routing header includes data previously read by the node sending theframe from the global memory address space in response to receiving aread routing header.
 7. The apparatus of claim 1, wherein the protocolengine is configured to instruct the parallel link input/output port toprepend the routing header onto the MAC frame and to write the frame tothe global memory address space that is physically located in the randomaccess memory device of another node.
 8. The apparatus of claim 1,wherein the protocol engine is configured to write the frame to theglobal memory address space that is physically located in the randomaccess memory device of another node based on a node ID associated withthe other node included in the routing header of the frame.
 9. Theapparatus of claim 1, wherein the one or more CPUs of each node areconfigured to write the frames to the local memory address space of thenode, wherein each of the frames includes a header that includes adestination MAC address and a source MAC address.
 10. The apparatus ofclaim 1, further comprising a switch configured to write the framesbetween the parallel link input/output ports based on the routingheader.
 11. The apparatus of claim 1, further comprising: a switchconfigured to route the frames between the parallel link input/outputports based on a routing header prepended onto the frames by a packetmanager; wherein the parallel link input/output ports are eachconfigured to prepend routing headers onto the frames.
 12. An apparatuscomprising: a plurality of nodes, each node including: a first centralprocessing unit (CPU) configured to write a frame to its own localmemory address space but not to the local memory address space of theother nodes; a memory divided between the local memory address space anda global memory address space associated with a global address; and aprotocol engine configured to: instruct a parallel link input/outputport to prepend a routing header onto the frame stored in the localmemory address space; send the frame out of the node through a parallellink input/output port; and receive frames through the parallel linkinput/output port, and write the frame to global memory address space ofthe node based on the routing header including the global addressassociated with the global memory address space; wherein the parallellink input/output port configured to: prepend a routing header onto theframe stored in the local memory address space before sending the frameto a switch; and strip the routing header from the frame after receivingthe frame from the switch; and route the frame between the protocolengine and the switch; and the switch configured to route the framebetween the plurality of nodes based on the routing header.
 13. Theapparatus of claim 12, wherein the first CPU, the memory, and theprotocol engine of each node are coupled to each other via a bus. 14.The apparatus of claim 12, wherein each node further includes a secondcentral processing unit (CPU) configured to write the frame to the localmemory address space.
 15. The apparatus of claim 12, wherein the firstCPU is configured to write the frame to the local memory space, theframe including a destination MAC address and a source MAC address. 16.The apparatus of claim 12, wherein each node is associated with a MACaddress and an Internet Protocol address.
 17. The apparatus of claim 12,wherein the protocol engine includes a direct memory access (DMA)engine.
 18. The apparatus of claim 12, wherein: each node furtherincludes an Ethernet port; and the apparatus further includes anEthernet switch configured to route data between the Ethernet ports ofeach node.
 19. The apparatus of claim 12, wherein each of the pluralityof nodes has a distributed memory manager loaded thereon, thedistributed memory manager being configured to partition the memory intothe local memory accessible only to the node of which the memory is acomponent and the global memory accessible to all of the plurality ofnodes in the apparatus.
 20. The apparatus of claim 12, wherein theprepended routing header includes a command type field indicating that atype of the frame is one of packet, write, read, or read response.